home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
SGI Freeware 2001 May
/
SGI Freeware 2001 May - Disc 2.iso
/
dist
/
fw_bzip2.idb
/
usr
/
freeware
/
info
/
bzip2.info-1.z
/
bzip2.info-1
(
.txt
)
Wrap
GNU Info File
|
2000-06-09
|
46KB
|
862 lines
This is Info file bzip2.info, produced by Makeinfo version 1.68 from
the input file manual.texi.
START-INFO-DIR-ENTRY
* Bzip2: (bzip2). A program and library for data compression.
END-INFO-DIR-ENTRY
File: bzip2.info, Node: Top, Next: Overview, Prev: (dir), Up: (dir)
This program, `bzip2', and associated library `libbzip2', are
Copyright (C) 1996-1999 Julian R Seward. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* The origin of this software must not be misrepresented; you must
not claim that you wrote the original software. If you use this
software in a product, an acknowledgment in the product
documentation would be appreciated but is not required.
* Altered source versions must be plainly marked as such, and must
not be misrepresented as being the original software.
* The name of the author may not be used to endorse or promote
products derived from this software without specific prior written
permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
Julian Seward, Cambridge, UK.
`jseward@acm.org'
`http://www.muraroa.demon.co.uk'
`bzip2'/`libbzip2' version 0.9.5 of 24 May 1999.
PATENTS: To the best of my knowledge, `bzip2' does not use any
patented algorithms. However, I do not have the resources available to
carry out a full patent search. Therefore I cannot give any guarantee
of the above statement.
* Menu:
* Overview:: An overview of bzip2
* How to use bzip2:: Invoking bzip2
* Programming with libbzip2:: Using the libbz library
* Miscellanea:: Random thoughts
-- The Detailed Node Listing --
Programming with `libbzip2'
* Top-level structure::
* Error handling::
* Low-level interface::
* High-level interface::
* Utility functions::
* zlib compatibility functions::
* Using the library in a stdio-free environment::
* Making a Windows DLL::
Miscellanea
* Limitations of the compressed file format::
* Portability issues::
* Reporting bugs::
* Did you get the right package?::
* Testing::
* Further reading::
File: bzip2.info, Node: Overview, Next: How to use bzip2, Prev: Top, Up: Top
Introduction
************
`bzip2' compresses files using the Burrows-Wheeler block-sorting
text compression algorithm, and Huffman coding. Compression is
generally considerably better than that achieved by more conventional
LZ77/LZ78-based compressors, and approaches the performance of the
PPM family of statistical compressors.
`bzip2' is built on top of `libbzip2', a flexible library for
handling compressed data in the `bzip2' format. This manual describes
both how to use the program and how to work with the library interface.
Most of the manual is devoted to this library, not the program, which
is good news if your interest is only in the program.
Chapter 2 describes how to use `bzip2'; this is the only part you
need to read if you just want to know how to operate the program.
Chapter 3 describes the programming interfaces in detail, and Chapter 4
records some miscellaneous notes which I thought ought to be recorded
somewhere.
* Menu:
* How to use bzip2::
* Programming with libbzip2::
* Miscellanea::
File: bzip2.info, Node: How to use bzip2, Next: Programming with libbzip2, Prev: Overview, Up: Top
How to use `bzip2'
******************
This chapter contains a copy of the `bzip2' man page, and nothing
else.
NAME
* `bzip2', `bunzip2' - a block-sorting file compressor, v0.9.5
* `bzcat' - decompresses files to stdout
* `bzip2recover' - recovers data from damaged bzip2 files
SYNOPSIS
........
* `bzip2' [ -cdfkqstvzVL123456789 ] [ filenames ... ]
* `bunzip2' [ -fkvsVL ] [ filenames ... ]
* `bzcat' [ -s ] [ filenames ... ]
* `bzip2recover' filename
DESCRIPTION
...........
`bzip2' compresses files using the Burrows-Wheeler block sorting
text compression algorithm, and Huffman coding. Compression is
generally considerably better than that achieved by more conventional
LZ77/LZ78-based compressors, and approaches the performance of the PPM
family of statistical compressors.
The command-line options are deliberately very similar to those of
GNU `gzip', but they are not identical.
`bzip2' expects a list of file names to accompany the command-line
flags. Each file is replaced by a compressed version of itself, with
the name `original_name.bz2'. Each compressed file has the same
modification date, permissions, and, when possible, ownership as the
corresponding original, so that these properties can be correctly
restored at decompression time. File name handling is naive in the
sense that there is no mechanism for preserving original file names,
permissions, ownerships or dates in filesystems which lack these
concepts, or have serious file name length restrictions, such as MS-DOS.
`bzip2' and `bunzip2' will by default not overwrite existing
files. If you want this to happen, specify the `-f' flag.
If no file names are specified, `bzip2' compresses from standard
input to standard output. In this case, `bzip2' will decline to write
compressed output to a terminal, as this would be entirely
incomprehensible and therefore pointless.
`bunzip2' (or `bzip2 -d') decompresses all specified files. Files
which were not created by `bzip2' will be detected and ignored, and a
warning issued. `bzip2' attempts to guess the filename for the
decompressed file from that of the compressed file as follows:
* `filename.bz2 ' becomes `filename'
* `filename.bz ' becomes `filename'
* `filename.tbz2' becomes `filename.tar'
* `filename.tbz ' becomes `filename.tar'
* `anyothername ' becomes `anyothername.out' If the file does
not end in one of the recognised endings, `.bz2', `.bz', `.tbz2' or
`.tbz', `bzip2' complains that it cannot guess the name of the original
file, and uses the original name with `.out' appended.
As with compression, supplying no filenames causes decompression
from standard input to standard output.
`bunzip2' will correctly decompress a file which is the
concatenation of two or more compressed files. The result is the
concatenation of the corresponding uncompressed files. Integrity
testing (`-t') of concatenated compressed files is also supported.
You can also compress or decompress files to the standard output by
giving the `-c' flag. Multiple files may be compressed and
decompressed like this. The resulting outputs are fed sequentially to
stdout. Compression of multiple files in this manner generates a stream
containing multiple compressed file representations. Such a stream can
be decompressed correctly only by `bzip2' version 0.9.0 or later.
Earlier versions of `bzip2' will stop after decompressing the first
file in the stream.
`bzcat' (or `bzip2 -dc') decompresses all specified files to the
standard output.
`bzip2' will read arguments from the environment variables `BZIP2'
and `BZIP', in that order, and will process them before any arguments
read from the command line. This gives a convenient way to supply
default arguments.
Compression is always performed, even if the compressed file is
slightly larger than the original. Files of less than about one
hundred bytes tend to get larger, since the compression mechanism has a
constant overhead in the region of 50 bytes. Random data (including
the output of most file compressors) is coded at about 8.05 bits per
byte, giving an expansion of around 0.5%.
As a self-check for your protection, `bzip2' uses 32-bit CRCs to
make sure that the decompressed version of a file is identical to the
original. This guards against corruption of the compressed data, and
against undetected bugs in `bzip2' (hopefully very unlikely). The
chances of data corruption going undetected is microscopic, about one
chance in four billion for each file processed. Be aware, though, that
the check occurs upon decompression, so it can only tell you that
something is wrong. It can't help you recover the original uncompressed
data. You can use `bzip2recover' to try to recover data from damaged
files.
Return values: 0 for a normal exit, 1 for environmental problems
(file not found, invalid flags, I/O errors, &c), 2 to indicate a corrupt
compressed file, 3 for an internal consistency error (eg, bug) which
caused `bzip2' to panic.
OPTIONS
.......
`-c --stdout'
Compress or decompress to standard output.
`-d --decompress'
Force decompression. `bzip2', `bunzip2' and `bzcat' are
really the same program, and the decision about what actions
to take is done on the basis of which name is used. This
flag overrides that mechanism, and forces bzip2 to decompress.
`-z --compress'
The complement to `-d': forces compression, regardless of the
invokation name.
`-t --test'
Check integrity of the specified file(s), but don't
decompress them. This really performs a trial decompression
and throws away the result.
`-f --force'
Force overwrite of output files. Normally, `bzip2' will not
overwrite existing output files. Also forces `bzip2' to
break hard links to files, which it otherwise wouldn't do.
`-k --keep'
Keep (don't delete) input files during compression or
decompression.
`-s --small'
Reduce memory usage, for compression, decompression and
testing. Files are decompressed and tested using a modified
algorithm which only requires 2.5 bytes per block byte. This
means any file can be decompressed in 2300k of memory, albeit
at about half the normal speed.
During compression, `-s' selects a block size of 200k, which
limits memory use to around the same figure, at the expense
of your compression ratio. In short, if your machine is low
on memory (8 megabytes or less), use -s for everything. See
MEMORY MANAGEMENT below.
`-q --quiet'
Suppress non-essential warning messages. Messages pertaining
to I/O errors and other critical events will not be
suppressed.
`-v --verbose'
Verbose mode - show the compression ratio for each file
processed. Further `-v''s increase the verbosity level,
spewing out lots of information which is primarily of
interest for diagnostic purposes.
`-L --license -V --version'
Display the software version, license terms and conditions.
`-1 to -9'
Set the block size to 100 k, 200 k .. 900 k when
compressing. Has no effect when decompressing. See MEMORY
MANAGEMENT below.
`--'
Treats all subsequent arguments as file names, even if they
start with a dash. This is so you can handle files with
names beginning with a dash, for example: `bzip2 --
-myfilename'.
`--repetitive-fast'
`--repetitive-best'
These flags are redundant in versions 0.9.5 and above. They
provided some coarse control over the behaviour of the
sorting algorithm in earlier versions, which was sometimes
useful. 0.9.5 and above have an improved algorithm which
renders these flags irrelevant.
MEMORY MANAGEMENT
.................
`bzip2' compresses large files in blocks. The block size affects
both the compression ratio achieved, and the amount of memory needed for
compression and decompression. The flags `-1' through `-9' specify the
block size to be 100,000 bytes through 900,000 bytes (the default)
respectively. At decompression time, the block size used for
compression is read from the header of the compressed file, and
`bunzip2' then allocates itself just enough memory to decompress the
file. Since block sizes are stored in compressed files, it follows
that the flags `-1' to `-9' are irrelevant to and so ignored during
decompression.
Compression and decompression requirements, in bytes, can be
estimated as:
Compression: 400k + ( 8 x block size )
Decompression: 100k + ( 4 x block size ), or
100k + ( 2.5 x block size )
Larger block sizes give rapidly diminishing marginal returns.
Most of the compression comes from the first two or three hundred k of
block size, a fact worth bearing in mind when using `bzip2' on small
machines. It is also important to appreciate that the decompression
memory requirement is set at compression time by the choice of block
size.
For files compressed with the default 900k block size, `bunzip2'
will require about 3700 kbytes to decompress. To support decompression
of any file on a 4 megabyte machine, `bunzip2' has an option to
decompress using approximately half this amount of memory, about 2300
kbytes. Decompression speed is also halved, so you should use this
option only where necessary. The relevant flag is `-s'.
In general, try and use the largest block size memory constraints
allow, since that maximises the compression achieved. Compression and
decompression speed are virtually unaffected by block size.
Another significant point applies to files which fit in a single
block - that means most files you'd encounter using a large block size.
The amount of real memory touched is proportional to the size of the
file, since the file is smaller than a block. For example, compressing
a file 20,000 bytes long with the flag `-9' will cause the compressor to
allocate around 7600k of memory, but only touch 400k + 20000 * 8 = 560
kbytes of it. Similarly, the decompressor will allocate 3700k but only
touch 100k + 20000 * 4 = 180 kbytes.
Here is a table which summarises the maximum memory usage for
different block sizes. Also recorded is the total compressed size for
14 files of the Calgary Text Compression Corpus totalling 3,141,622
bytes. This column gives some feel for how compression varies with
block size. These figures tend to understate the advantage of larger
block sizes for larger files, since the Corpus is dominated by smaller
files.
Compress Decompress Decompress Corpus
Flag usage usage -s usage Size
-1 1200k 500k 350k 914704
-2 2000k 900k 600k 877703
-3 2800k 1300k 850k 860338
-4 3600k 1700k 1100k 846899
-5 4400k 2100k 1350k 845160
-6 5200k 2500k 1600k 838626
-7 6100k 2900k 1850k 834096
-8 6800k 3300k 2100k 828642
-9 7600k 3700k 2350k 828642
RECOVERING DATA FROM DAMAGED FILES
..................................
`bzip2' compresses files in blocks, usually 900kbytes long. Each
block is handled independently. If a media or transmission error causes
a multi-block `.bz2' file to become damaged, it may be possible to
recover data from the undamaged blocks in the file.
The compressed representation of each block is delimited by a
48-bit pattern, which makes it possible to find the block boundaries
with reasonable certainty. Each block also carries its own 32-bit CRC,
so damaged blocks can be distinguished from undamaged ones.
`bzip2recover' is a simple program whose purpose is to search for
blocks in `.bz2' files, and write each block out into its own `.bz2'
file. You can then use `bzip2 -t' to test the integrity of the
resulting files, and decompress those which are undamaged.
`bzip2recover' takes a single argument, the name of the damaged
file, and writes a number of files `rec0001file.bz2',
`rec0002file.bz2', etc, containing the extracted blocks. The
output filenames are designed so that the use of wildcards
in subsequent processing - for example, `bzip2 -dc rec*file.bz2 >
recovered_data' - lists the files in the correct order.
`bzip2recover' should be of most use dealing with large `.bz2'
files, as these will contain many blocks. It is clearly
futile to use it on damaged single-block files, since a
damaged block cannot be recovered. If you wish to minimise any
potential data loss through media or transmission errors, you might
consider compressing with a smaller block size.
PERFORMANCE NOTES
.................
The sorting phase of compression gathers together similar strings
in the file. Because of this, files containing very long runs of
repeated symbols, like "aabaabaabaab ..." (repeated several hundred
times) may compress more slowly than normal. Versions 0.9.5 and above
fare much better than previous versions in this respect. The ratio
between worst-case and average-case compression time is in the region
of 10:1. For previous versions, this figure was more like 100:1. You
can use the `-vvvv' option to monitor progress in great detail, if you
want.
Decompression speed is unaffected by these phenomena.
`bzip2' usually allocates several megabytes of memory to operate
in, and then charges all over it in a fairly random fashion. This means
that performance, both for compressing and decompressing, is largely
determined by the speed at which your machine can service cache misses.
Because of this, small changes to the code to reduce the miss rate have
been observed to give disproportionately large performance improvements.
I imagine `bzip2' will perform best on machines with very large caches.
CAVEATS
.......
I/O error messages are not as helpful as they could be. `bzip2'
tries hard to detect I/O errors and exit cleanly, but the details of
what the problem is sometimes seem rather misleading.
This manual page pertains to version 0.9.5 of `bzip2'. Compressed
data created by this version is entirely forwards and backwards
compatible with the previous public releases, versions 0.1pl2 and 0.9.0,
but with the following exception: 0.9.0 and above can correctly
decompress multiple concatenated compressed files. 0.1pl2 cannot do
this; it will stop after decompressing just the first file in the
stream.
`bzip2recover' uses 32-bit integers to represent bit positions in
compressed files, so it cannot handle compressed files more than 512
megabytes long. This could easily be fixed.
AUTHOR
......
Julian Seward, `jseward@acm.org'.
The ideas embodied in `bzip2' are due to (at least) the following
people: Michael Burrows and David Wheeler (for the block sorting
transformation), David Wheeler (again, for the Huffman coder), Peter
Fenwick (for the structured coding model in the original `bzip', and
many refinements), and Alistair Moffat, Radford Neal and Ian Witten
(for the arithmetic coder in the original `bzip'). I am much indebted
for their help, support and advice. See the manual in the source
distribution for pointers to sources of documentation. Christian von
Roques encouraged me to look for faster sorting algorithms, so as to
speed up compression. Bela Lubkin encouraged me to improve the
worst-case compression performance. Many people sent patches, helped
with portability problems, lent machines, gave advice and were generally
helpful.
File: bzip2.info, Node: Programming with libbzip2, Next: Miscellanea, Prev: How to use bzip2, Up: Top
Programming with `libbzip2'
***************************
This chapter describes the programming interface to `libbzip2'.
For general background information, particularly about memory use
and performance aspects, you'd be well advised to read Chapter 2 as
well.
* Menu:
* Top-level structure:: Functions and data structures
* Error handling::
* Low-level interface::
* High-level interface::
* Utility functions::
* zlib compatibility functions::
* Using the library in a stdio-free environment::
* Making a Windows DLL::
File: bzip2.info, Node: Top-level structure, Next: Error handling, Prev: Programming with libbzip2, Up: Programming with libbzip2
Top-level structure
===================
`libbzip2' is a flexible library for compressing and decompressing
data in the `bzip2' data format. Although packaged as a single entity,
it helps to regard the library as three separate parts: the low level
interface, and the high level interface, and some utility functions.
The structure of `libbzip2''s interfaces is similar to that of
Jean-loup Gailly's and Mark Adler's excellent `zlib' library.
Low-level summary
-----------------
This interface provides services for compressing and decompressing
data in memory. There's no provision for dealing with files, streams
or any other I/O mechanisms, just straight memory-to-memory work. In
fact, this part of the library can be compiled without inclusion of
`stdio.h', which may be helpful for embedded applications.
The low-level part of the library has no global variables and is
therefore thread-safe.
Six routines make up the low level interface: `bzCompressInit',
`bzCompress', and
`bzCompressEnd' for compression, and a corresponding trio
`bzDecompressInit',
`bzDecompress' and `bzDecompressEnd' for decompression. The `*Init'
functions allocate memory for compression/decompression and do other
initialisations, whilst the `*End' functions close down operations and
release memory.
The real work is done by `bzCompress' and `bzDecompress'. These
compress/decompress data from a user-supplied input buffer to a
user-supplied output buffer. These buffers can be any size; arbitrary
quantities of data are handled by making repeated calls to these
functions. This is a flexible mechanism allowing a consumer-pull style
of activity, or producer-push, or a mixture of both.
High-level summary
------------------
This interface provides some handy wrappers around the low-level
interface to facilitate reading and writing `bzip2' format files
(`.bz2' files). The routines provide hooks to facilitate reading files
in which the `bzip2' data stream is embedded within some larger-scale
file structure, or where there are multiple `bzip2' data streams
concatenated end-to-end.
For reading files, `bzReadOpen', `bzRead', `bzReadClose' and
`bzReadGetUnused' are supplied. For writing files, `bzWriteOpen',
`bzWrite' and `bzWriteFinish' are available.
As with the low-level library, no global variables are used so the
library is per se thread-safe. However, if I/O errors occur whilst
reading or writing the underlying compressed files, you may have to
consult `errno' to determine the cause of the error. In that case,
you'd need a C library which correctly supports `errno' in a
multithreaded environment.
To make the library a little simpler and more portable, `bzReadOpen'
and `bzWriteOpen' require you to pass them file handles (`FILE*'s)
which have previously been opened for reading or writing respectively.
That avoids portability problems associated with file operations and
file attributes, whilst not being much of an imposition on the
programmer.
Utility functions summary
-------------------------
For very simple needs, `bzBuffToBuffCompress' and
`bzBuffToBuffDecompress' are provided. These compress data in memory
from one buffer to another buffer in a single function call. You
should assess whether these functions fulfill your memory-to-memory
compression/decompression requirements before investing effort in
understanding the more general but more complex low-level interface.
Yoshioka Tsuneo (`QWF00133@niftyserve.or.jp' /
`tsuneo-y@is.aist-nara.ac.jp') has contributed some functions to give
better `zlib' compatibility. These functions are `bzopen', `bzread',
`bzwrite', `bzflush', `bzclose', `bzerror' and `bzlibVersion'. You may
find these functions more convenient for simple file reading and
writing, than those in the high-level interface. These functions are
not (yet) officially part of the library, and are minimally documented
here. If they break, you get to keep all the pieces. I hope to
document them properly when time permits.
Yoshioka also contributed modifications to allow the library to be
built as a Windows DLL.
File: bzip2.info, Node: Error handling, Next: Low-level interface, Prev: Top-level structure, Up: Programming with libbzip2
Error handling
==============
The library is designed to recover cleanly in all situations,
including the worst-case situation of decompressing random data. I'm
not 100% sure that it can always do this, so you might want to add a
signal handler to catch segmentation violations during decompression if
you are feeling especially paranoid. I would be interested in hearing
more about the robustness of the library to corrupted compressed data.
The file `bzlib.h' contains all definitions needed to use the
library. In particular, you should definitely not include
`bzlib_private.h'.
In `bzlib.h', the various return values are defined. The following
list is not intended as an exhaustive description of the circumstances
in which a given value may be returned - those descriptions are given
later. Rather, it is intended to convey the rough meaning of each
return value. The first five actions are normal and not intended to
denote an error situation.
`BZ_OK'
The requested action was completed successfully.
`BZ_RUN_OK'
`BZ_FLUSH_OK'
`BZ_FINISH_OK'
In `bzCompress', the requested flush/finish/nothing-special action
was completed successfully.
`BZ_STREAM_END'
Compression of data was completed, or the logical stream end was
detected during decompression.
The following return values indicate an error of some kind.
`BZ_SEQUENCE_ERROR'
When using the library, it is important to call the functions in
the correct sequence and with data structures (buffers etc) in the
correct states. `libbzip2' checks as much as it can to ensure
this is happening, and returns `BZ_SEQUENCE_ERROR' if not. Code
which complies precisely with the function semantics, as detailed
below, should never receive this value; such an event denotes
buggy code which you should investigate.
`BZ_PARAM_ERROR'
Returned when a parameter to a function call is out of range or
otherwise manifestly incorrect. As with `BZ_SEQUENCE_ERROR', this
denotes a bug in the client code. The distinction between
`BZ_PARAM_ERROR' and `BZ_SEQUENCE_ERROR' is a bit hazy, but still
worth making.
`BZ_MEM_ERROR'
Returned when a request to allocate memory failed. Note that the
quantity of memory needed to decompress a stream cannot be
determined until the stream's header has been read. So
`bzDecompress' and `bzRead' may return `BZ_MEM_ERROR' even though
some of the compressed data has been read. The same is not true
for compression; once `bzCompressInit' or `bzWriteOpen' have
successfully completed, `BZ_MEM_ERROR' cannot occur.
`BZ_DATA_ERROR'
Returned when a data integrity error is detected during
decompression. Most importantly, this means when stored and
computed CRCs for the data do not match. This value is also
returned upon detection of any other anomaly in the compressed
data.
`BZ_DATA_ERROR_MAGIC'
As a special case of `BZ_DATA_ERROR', it is sometimes useful to
know when the compressed stream does not start with the correct
magic bytes (`'B' 'Z' 'h'').
`BZ_IO_ERROR'
Returned by `bzRead' and `bzRead' when there is an error reading
or writing in the compressed file, and by `bzReadOpen' and
`bzWriteOpen' for attempts to use a file for which the error
indicator (viz, `ferror(f)') is set. On receipt of `BZ_IO_ERROR',
the caller should consult `errno' and/or `perror' to acquire
operating-system specific information about the problem.
`BZ_UNEXPECTED_EOF'
Returned by `bzRead' when the compressed file finishes before the
logical end of stream is detected.
`BZ_OUTBUFF_FULL'
Returned by `bzBuffToBuffCompress' and `bzBuffToBuffDecompress' to
indicate that the output data will not fit into the output buffer
provided.
File: bzip2.info, Node: Low-level interface, Next: High-level interface, Prev: Error handling, Up: Programming with libbzip2
Low-level interface
===================
`bzCompressInit'
----------------
typedef
struct {
char *next_in;
unsigned int avail_in;
unsigned int total_in;
char *next_out;
unsigned int avail_out;
unsigned int total_out;
void *state;
void *(*bzalloc)(void *,int,int);
void (*bzfree)(void *,void *);
void *opaque;
}
bz_stream;
int bzCompressInit ( bz_stream *strm,
int blockSize100k,
int verbosity,
int workFactor );
Prepares for compression. The `bz_stream' structure holds all data
pertaining to the compression activity. A `bz_stream' structure should
be allocated and initialised prior to the call. The fields of
`bz_stream' comprise the entirety of the user-visible data. `state' is
a pointer to the private data structures required for compression.
Custom memory allocators are supported, via fields `bzalloc',
`bzfree', and `opaque'. The value `opaque' is passed to as the first
argument to all calls to `bzalloc' and `bzfree', but is otherwise
ignored by the library. The call `bzalloc ( opaque, n, m )' is
expected to return a pointer `p' to `n * m' bytes of memory, and
`bzfree ( opaque, p )' should free that memory.
If you don't want to use a custom memory allocator, set `bzalloc',
`bzfree' and `opaque' to `NULL', and the library will then use the
standard `malloc'/`free' routines.
Before calling `bzCompressInit', fields `bzalloc', `bzfree' and
`opaque' should be filled appropriately, as just described. Upon
return, the internal state will have been allocated and initialised,
and `total_in' and `total_out' will have been set to zero. These last
two fields are used by the library to inform the caller of the total
amount of data passed into and out of the library, respectively. You
should not try to change them.
Parameter `blockSize100k' specifies the block size to be used for
compression. It should be a value between 1 and 9 inclusive, and the
actual block size used is 100000 x this figure. 9 gives the best
compression but takes most memory.
Parameter `verbosity' should be set to a number between 0 and 4
inclusive. 0 is silent, and greater numbers give increasingly verbose
monitoring/debugging output. If the library has been compiled with
`-DBZ_NO_STDIO', no such output will appear for any verbosity setting.
Parameter `workFactor' controls how the compression phase behaves
when presented with worst case, highly repetitive, input data. If
compression runs into difficulties caused by repetitive data, the
library switches from the standard sorting algorithm to a fallback
algorithm. The fallback is slower than the standard algorithm by
perhaps a factor of three, but always behaves reasonably, no matter how
bad the input.
Lower values of `workFactor' reduce the amount of effort the
standard algorithm will expend before resorting to the fallback. You
should set this parameter carefully; too low, and many inputs will be
handled by the fallback algorithm and so compress rather slowly, too
high, and your average-to-worst case compression times can become very
large. The default value of 30 gives reasonable behaviour over a wide
range of circumstances.
Allowable values range from 0 to 250 inclusive. 0 is a special case,
equivalent to using the default value of 30.
Note that the compressed output generated is the same regardless of
whether or not the fallback algorithm is used.
Be aware also that this parameter may disappear entirely in future
versions of the library. In principle it should be possible to devise a
good way to automatically choose which algorithm to use. Such a
mechanism would render the parameter obsolete.
Possible return values:
`BZ_PARAM_ERROR'
if `strm' is `NULL'
or `blockSize' < 1 or `blockSize' > 9
or `verbosity' < 0 or `verbosity' > 4
or `workFactor' < 0 or `workFactor' > 250
`BZ_MEM_ERROR'
if not enough memory is available
`BZ_OK'
otherwise
Allowable next actions:
`bzCompress'
if `BZ_OK' is returned
no specific action needed in case of error
`bzCompress'
------------
int bzCompress ( bz_stream *strm, int action );
Provides more input and/or output buffer space for the library. The
caller maintains input and output buffers, and calls `bzCompress' to
transfer data between them.
Before each call to `bzCompress', `next_in' should point at the data
to be compressed, and `avail_in' should indicate how many bytes the
library may read. `bzCompress' updates `next_in', `avail_in' and
`total_in' to reflect the number of bytes it has read.
Similarly, `next_out' should point to a buffer in which the
compressed data is to be placed, with `avail_out' indicating how much
output space is available. `bzCompress' updates `next_out',
`avail_out' and `total_out' to reflect the number of bytes output.
You may provide and remove as little or as much data as you like on
each call of `bzCompress'. In the limit, it is acceptable to supply and
remove data one byte at a time, although this would be terribly
inefficient. You should always ensure that at least one byte of output
space is available at each call.
A second purpose of `bzCompress' is to request a change of mode of
the compressed stream.
Conceptually, a compressed stream can be in one of four states: IDLE,
RUNNING, FLUSHING and FINISHING. Before initialisation
(`bzCompressInit') and after termination (`bzCompressEnd'), a stream is
regarded as IDLE.
Upon initialisation (`bzCompressInit'), the stream is placed in the
RUNNING state. Subsequent calls to `bzCompress' should pass `BZ_RUN'
as the requested action; other actions are illegal and will result in
`BZ_SEQUENCE_ERROR'.
At some point, the calling program will have provided all the input
data it wants to. It will then want to finish up - in effect, asking
the library to process any data it might have buffered internally. In
this state, `bzCompress' will no longer attempt to read data from
`next_in', but it will want to write data to `next_out'. Because the
output buffer supplied by the user can be arbitrarily small, the
finishing-up operation cannot necessarily be done with a single call of
`bzCompress'.
Instead, the calling program passes `BZ_FINISH' as an action to
`bzCompress'. This changes the stream's state to FINISHING. Any
remaining input (ie, `next_in[0 .. avail_in-1]') is compressed and
transferred to the output buffer. To do this, `bzCompress' must be
called repeatedly until all the output has been consumed. At that
point, `bzCompress' returns `BZ_STREAM_END', and the stream's state is
set back to IDLE. `bzCompressEnd' should then be called.
Just to make sure the calling program does not cheat, the library
makes a note of `avail_in' at the time of the first call to
`bzCompress' which has `BZ_FINISH' as an action (ie, at the time the
program has announced its intention to not supply any more input). By
comparing this value with that of `avail_in' over subsequent calls to
`bzCompress', the library can detect any attempts to slip in more data
to compress. Any calls for which this is detected will return
`BZ_SEQUENCE_ERROR'. This indicates a programming mistake which should
be corrected.
Instead of asking to finish, the calling program may ask
`bzCompress' to take all the remaining input, compress it and terminate
the current (Burrows-Wheeler) compression block. This could be useful
for error control purposes. The mechanism is analogous to that for
finishing: call `bzCompress' with an action of `BZ_FLUSH', remove
output data, and persist with the `BZ_FLUSH' action until the value
`BZ_RUN' is returned. As with finishing, `bzCompress' detects any
attempt to provide more input data once the flush has begun.
Once the flush is complete, the stream returns to the normal RUNNING
state.
This all sounds pretty complex, but isn't really. Here's a table
which shows which actions are allowable in each state, what action will
be taken, what the next state is, and what the non-error return values
are. Note that you can't explicitly ask what state the stream is in,
but nor do you need to - it can be inferred from the values returned by
`bzCompress'.
IDLE/`any'
Illegal. IDLE state only exists after `bzCompressEnd' or
before `bzCompressInit'.
Return value = `BZ_SEQUENCE_ERROR'
RUNNING/`BZ_RUN'
Compress from `next_in' to `next_out' as much as possible.
Next state = RUNNING
Return value = `BZ_RUN_OK'
RUNNING/`BZ_FLUSH'
Remember current value of `next_in'. Compress from `next_in'
to `next_out' as much as possible, but do not accept any more input.
Next state = FLUSHING
Return value = `BZ_FLUSH_OK'
RUNNING/`BZ_FINISH'
Remember current value of `next_in'. Compress from `next_in'
to `next_out' as much as possible, but do not accept any more input.
Next state = FINISHING
Return value = `BZ_FINISH_OK'
FLUSHING/`BZ_FLUSH'
Compress from `next_in' to `next_out' as much as possible,
but do not accept any more input.
If all the existing input has been used up and all compressed
output has been removed
Next state = RUNNING; Return value = `BZ_RUN_OK'
else
Next state = FLUSHING; Return value = `BZ_FLUSH_OK'
FLUSHING/other
Illegal.
Return value = `BZ_SEQUENCE_ERROR'
FINISHING/`BZ_FINISH'
Compress from `next_in' to `next_out' as much as possible,
but to not accept any more input.
If all the existing input has been used up and all compressed
output has been removed
Next state = IDLE; Return value = `BZ_STREAM_END'
else
Next state = FINISHING; Return value = `BZ_FINISHING'
FINISHING/other
Illegal.
Return value = `BZ_SEQUENCE_ERROR'
That still looks complicated? Well, fair enough. The usual sequence
of calls for compressing a load of data is:
* Get started with `bzCompressInit'.
* Shovel data in and shlurp out its compressed form using zero or
more calls of `bzCompress' with action = `BZ_RUN'.
* Finish up. Repeatedly call `bzCompress' with action = `BZ_FINISH',
copying out the compressed output, until `BZ_STREAM_END' is
returned.
* Close up and go home. Call `bzCompressEnd'. If the data you want
to compress fits into your input buffer all at once, you can skip the
calls of `bzCompress ( ..., BZ_RUN )' and just do the `bzCompress (
..., BZ_FINISH )' calls.
All required memory is allocated by `bzCompressInit'. The
compression library can accept any data at all (obviously). So you
shouldn't get any error return values from the `bzCompress' calls. If
you do, they will be `BZ_SEQUENCE_ERROR', and indicate a bug in your
programming.
Trivial other possible return values:
`BZ_PARAM_ERROR'
if `strm' is `NULL', or `strm->s' is `NULL'
`bzCompressEnd'
---------------
int bzCompressEnd ( bz_stream *strm );
Releases all memory associated with a compression stream.
Possible return values:
`BZ_PARAM_ERROR' if `strm' is `NULL' or `strm->s' is `NULL'
`BZ_OK' otherwise
`bzDecompressInit'
------------------
int bzDecompressInit ( bz_stream *strm, int verbosity, int small );
Prepares for decompression. As with `bzCompressInit', a `bz_stream'
record should be allocated and initialised before the call. Fields
`bzalloc', `bzfree' and `opaque' should be set if a custom memory
allocator is required, or made `NULL' for the normal `malloc'/`free'
routines. Upon return, the internal state will have been initialised,
and `total_in' and `total_out' will be zero.
For the meaning of parameter `verbosity', see `bzCompressInit'.
If `small' is nonzero, the library will use an alternative
decompression algorithm which uses less memory but at the cost of
decompressing more slowly (roughly speaking, half the speed, but the
maximum memory requirement drops to around 2300k). See Chapter 2 for
more information on memory management.
Note that the amount of memory needed to decompress a stream cannot
be determined until the stream's header has been read, so even if
`bzDecompressInit' succeeds, a subsequent `bzDecompress' could fail
with `BZ_MEM_ERROR'.
Possible return values:
`BZ_PARAM_ERROR'
if `(small != 0 && small != 1)'
or `(verbosity < 0 || verbosity > 4)'
`BZ_MEM_ERROR'
if insufficient memory is available
Allowable next actions:
`bzDecompress'
if `BZ_OK' was returned
no specific action required in case of error
`bzDecompress'
--------------
int bzDecompress ( bz_stream *strm );
Provides more input and/out output buffer space for the library. The
caller maintains input and output buffers, and uses `bzDecompress' to
transfer data between them.
Before each call to `bzDecompress', `next_in' should point at the
compressed data, and `avail_in' should indicate how many bytes the
library may read. `bzDecompress' updates `next_in', `avail_in' and
`total_in' to reflect the number of bytes it has read.
Similarly, `next_out' should point to a buffer in which the
uncompressed output is to be placed, with `avail_out' indicating how
much output space is available. `bzCompress' updates `next_out',
`avail_out' and `total_out' to reflect the number of bytes output.
You may provide and remove as little or as much data as you like on
each call of `bzDecompress'. In the limit, it is acceptable to supply
and remove data one byte at a time, although this would be terribly
inefficient. You should always ensure that at least one byte of output
space is available at each call.
Use of `bzDecompress' is simpler than `bzCompress'.
You should provide input and remove output as described above, and
repeatedly call `bzDecompress' until `BZ_STREAM_END' is returned.
Appearance of `BZ_STREAM_END' denotes that `bzDecompress' has detected
the logical end of the compressed stream. `bzDecompress' will not
produce `BZ_STREAM_END' until all output data has been placed into the
output buffer, so once `BZ_STREAM_END' appears, you are guaranteed to
have available all the decompressed output, and `bzDecompressEnd' can
safely be called.
If case of an error return value, you should call `bzDecompressEnd'
to clean up and release memory.
Possible return values:
`BZ_PARAM_ERROR'
if `strm' is `NULL' or `strm->s' is `NULL'
or `strm->avail_out < 1'
`BZ_DATA_ERROR'
if a data integrity error is detected in the compressed stream
`BZ_DATA_ERROR_MAGIC'
if the compressed stream doesn't begin with the right magic bytes
`BZ_MEM_ERROR'
if there wasn't enough memory available
`BZ_STREAM_END'
if the logical end of the data stream was detected and all
output in has been consumed, eg `s->avail_out > 0'
`BZ_OK'
otherwise
Allowable next actions:
`bzDecompress'
if `BZ_OK' was returned
`bzDecompressEnd'
otherwise
`bzDecompressEnd'
-----------------
int bzDecompressEnd ( bz_stream *strm );
Releases all memory associated with a decompression stream.
Possible return values:
`BZ_PARAM_ERROR'
if `strm' is `NULL' or `strm->s' is `NULL'
`BZ_OK'
otherwise
Allowable next actions:
None.